Learning in Complex Environments through Multiple Adaptive Partitions

When using tabular value functions, the application of Reinforcement Learning (RL) algorithms to real-world problems may have prohibitive memory requirements and learning time. In this paper, we introduce LEAP (Learning Entities Adaptive Partitioning), a novel model-free learning algorithm in which the state space is decomposed into several overlapping partitions which are dynamically modified to learn near-optimal policies with a small number of parameters. Starting from large macrostates, LEAP generates more refined partitions whenever it detects an incoherence between what has been learned and the actual return from the environment. Since in highly stochastic problems the adaptive process can lead to overrefinement, we introduce a mechanism that prunes the macrostates without affecting the learned policy. Through refinement and pruning, LEAP builds a multi-resolution state representation that is specialized only where it is actually needed. The learning properties of LEAP are verified in two experiments from [12].